Camera Augmented Reality Based Activity History Tracking

ABSTRACT

Augmented reality can be used to display previously captured images on a viewfinder of a camera as the camera&#39;s active position nears a position from which the picture was originally taken. A histogram file may associate the original image with positional information of the camera when the image was captured. When the cameras active position nears those coordinates, a transparent version of the digital image is displayed on the viewfinder of the camera. The positional information may include a spatial location of the camera (e.g., GPS coordinates, etc.) as well as an orientation of the camera (e.g., yaw, pitch, roll, etc.). Augmented reality can be used to guide the user to configure/re-configure the camera in order to correct (or avoid) an image quality issue/defect when re-taking a picture.

This patent application is a continuation of U.S. Non-Provisionalapplication Ser. No. 15/289,698 filed on Oct. 10, 2016 and entitled“Camera Augmented Reality Based Activity History Tracking,” which is acontinuation of U.S. Non-Provisional application Ser. No. 14/212,157filed on Mar. 14, 2014 and entitled “Camera Augmented Reality BasedActivity History Tracking,” which claims priority to U.S. ProvisionalApplication No. 61/784,667, filed on Mar. 14, 2013 and entitled “CameraAugmented Reality Based Activity History Tracking,” all of which arehereby incorporated by reference herein as if reproduced in theirentireties.

TECHNICAL FIELD

The present invention relates to a system and method for visual mediasystems, and, in particular embodiments, to techniques for cameraaugmented reality based activity history tracking.

BACKGROUND

Various factors and/or settings can influence image quality when takingpictures or videos with modern digital cameras. Indeed, some featuresand settings may be adjusted by the user to enhance image quality. Forinstance, exposure related parameters (e.g., aperture, shutter speed,ISO speed, etc.) may be optimized for various lighting conditions.Further, a camera position can be shifted to improve image quality, asmay be the case when a shadow or glare produces an image artifact.Novice users may lack the advanced knowledge/skill required toeffectively manipulate camera settings and/or camera positioning toachieve optimal image quality. Accordingly, mechanisms for directingusers to adjust camera settings and/or camera positioning are desired.

SUMMARY OF THE INVENTION

Technical advantages are generally achieved, by embodiments of thisdisclosure which describe techniques for camera augmented reality basedactivity history tracking.

In accordance with an embodiment, a method for operating a camera isprovided. In this example, the method comprises taking a picture with acamera to capture a first digital image, and detecting a position of thecamera when capturing the first digital image. The position of thecamera is associated with the first digital image in a histogram file.The method further includes monitoring an active position of the cameraafter capturing the first digital image, and displaying the firstdigital image on a viewfinder of the camera when the active position ofthe camera is within a threshold of the position specified by thehistogram file. An apparatus for performing this method is alsoprovided.

In accordance with another embodiment, a method for correcting imagequality in digital photography is provided. In this example, the methodcomprises taking a picture using a camera to obtain a first digitalimage, detecting an image quality problem in the first digital image,determining that the image quality problem is at least partiallyattributable to a configuration of the camera, instructing the user toreconfigure the camera via augmented reality, and re-taking the picturefollowing re-configuration of the camera to obtain a second digitalimage. The image quality problem is at least partially corrected in thesecond digital image. An apparatus for performing this method is alsoprovided.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the present invention, and theadvantages thereof, reference is now made to the following descriptionstaken in conjunction with the accompanying drawing, in which:

FIG. 1. illustrates a block diagram of a camera;

FIG. 2 illustrates a graph of a three dimensional map of cameraactivity;

FIG. 3 illustrates a diagram of an activity history map;

FIGS. 4A-4E illustrate diagrams of embodiment viewfinder images used toguide a user to re-align a camera prior to re-taking a picture;

FIG. 5 illustrates a flowchart of an embodiment method for usingaugmented reality to display a transparent version of a digital image ona viewfinder of a camera as the camera is returned to a position fromwhich the digital image was captured;

FIG. 6 illustrates a diagram of another embodiment viewfinder image usedto guide a user to re-align a camera prior to re-taking a picture viaaugmented reality;

FIG. 7 illustrates a diagram of yet another embodiment viewfinder imageused to guide a user to correct an image quality issue via augmentedreality;

FIG. 8 illustrates a flowchart of an embodiment method for correctingimage quality problems via augmented reality;

FIG. 9 illustrates a diagram of an embodiment viewfinder image of ahistogram file that associates a sequence of locations with a sequenceof digital images;

FIG. 10 illustrates a diagram of yet another embodiment viewfinder imagethat allows a user to cycle through a sequence of digital images;

FIG. 11 illustrates a flowchart of an embodiment method for displayingdigital images as a user re-traces locations in which those digitalimages were previously captured; and

FIG. 12 illustrates an embodiment of a block diagram of a processingsystem that may be used for implementing the devices and methodsdisclosed herein.

Corresponding numerals and symbols in the different figures generallyrefer to corresponding parts unless otherwise indicated. The figures aredrawn to clearly illustrate the relevant aspects of the embodiments andare not necessarily drawn to scale.

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

The making and using of the presently preferred embodiments arediscussed in detail below. It should be appreciated, however, that thepresent invention provides many applicable inventive concepts that canbe embodied in a wide variety of specific contexts. The specificembodiments discussed are merely illustrative of specific ways to makeand use the invention, and do not limit the scope of the invention.

Aspects of this disclosure use augmented reality to display previouslycaptured images on a viewfinder of a camera as the camera's activeposition nears a position from which the picture was originally taken.In some embodiments, positional information (e.g., coordinates, etc.) ofthe camera is associated with a digital image in a histogram file, and atransparent version of the digital image is displayed on the viewfinderof the camera when the camera is returned to that position. Thepositional information may include a spatial location of the camera(e.g., GPS coordinates, etc.) as well as an orientation of the camera(e.g., yaw, pitch, roll, etc.). Hence, pictures captured at the samelocation (but different angles) may be distinguished from one anotherbased on the camera's orientation. In some embodiments, augmentedreality is used to guide the user to configure/re-configure the camerain order to correct (or avoid) an image quality issue/defect whentaking/re-taking a picture. For example, an indication may be displayedon the viewfinder of the camera that instructs the user re-configure thecamera. The indication may instruct the user to re-align the camera, toadjust a camera setting (e.g., to change aperture/shutter setting,etc.), or to take the picture under different lighting conditions. Theseand other aspects are described in greater detail below.

FIG. 1 illustrates a camera 100 comprising a user interface 110, anoptical instrument module 120, and a sensors module 130. The userinterface 110 includes a viewfinder 112 and an input module 114. Theviewfinder 112 may be any component that allows the user to view apicture scene when composing a picture. For example, the viewfinder 112may be an LCD display on a camera phone. The input module 114 may be anycomponent that allows the user to manipulate the camera 100, such as akey pad or touchscreen. The optical instrument module 120 may includeany collection of components used to capture, process, and store adigital image. The optical instrument module 120 may include a centralprocessing unit (CPU) 122, a memory 124, a graphics processing unite(GPU) 126, a camera subsystem 128, and a sensors subsystem 130. The CPU122 may be any component capable of performing computations and/or otherprocessing related tasks, and the memory 124 may be any componentcapable of storing programming and/or instructions for the CPU 122. TheGPU 126 may be any component or collection of components configured tocreate an image in a memory buffer for output to a display, e.g., theviewfinder 112, etc. The camera subsystem 128 may include any componentor collection of components used to capture a digital image, e.g.,electronic sensor, shutter, etc. The sensors module 130 may include anycomponent or collection of components used to detect or monitor a cameraposition (e.g., location, orientation, etc.) or environmental condition(e.g., lighting, etc.). For example, the sensor module 130 may include agyroscope and/or accelerometer for detecting an orientation (e.g., yaw,pitch, roll, etc.) of the camera 100, a global positioning system (GPS)for detecting a spatial location of the camera 100, and a meteringsystem (e.g., ambient light sensor) for detecting environmental/lightinginformation. In some embodiments, the GPS may be a micro or indoor GPScapable of providing precise camera location, e.g., three dimensionalcoordinate information.

FIG. 2 illustrates a graph depicting a position of a camera 280. Asshown, the camera 280 has a spatial location (x, y, z) as well as aspatial orientation (roll, yaw, and pitch). The terms roll, yaw, andpitch are used loosely in this disclosure, and may correspond toelemental rotations, e.g., Euler angles (α, β, γ) about axes of a fixedor rotating coordinate system. In some embodiments, camera positions areassociated with digital images in an activity history map stored in ahistogram file. FIG. 3. Illustrates an activity history map 300associating sequence of digital images 301-308 with vectors along acoordinate plane. The point of each vector represents the spatiallocation (e.g., x, y, z) of the camera when the corresponding digitalimage was captured, while the arrow head represents the orientation ofthe camera when the corresponding digital image was captured. Forexample, the digital images 303-305 were captured in the same (orsimilar) spatial location, but at different camera orientations.

Aspects of this disclosure use augmented reality to display atransparent version of a digital image on the viewfinder of the camerawhen the camera nears a position from which the digital image waspreviously captured. FIGS. 4A-4E illustrate how digital images 410-460are displayed on a viewfinder 400 of a camera using augmented reality.FIG. 4A illustrates a viewfinder 400 of the camera before any images aredisplayed. FIG. 4B illustrates how a transparent version of a digitalimage 410 is displayed on the viewfinder 400 as an active position ofthe camera nears a position associated with the digital image 410 in ahistogram file. FIG. 4C illustrates how a transparent version of adigital image 420 is displayed on the viewfinder 400 as an orientationof the camera nears a spatial orientation associated with the digitalimage 420.

In some embodiments, the camera may include components (e.g., micro GPS,etc.) that are capable of detecting the location and/or orientation ofthe camera with high precision. In those embodiments, the transparentversion of the digital images 410, 420 may move gradually over theviewfinder 400 as the active position of the camera changes in relationto the position stored in the histogram file. For example, a differenceor shift (e.g., Δy, Δx) between the digital image 410 and a livepicture/scene displayed on the viewfinder 400 may decrease as the activelocation of the camera approaches the location stored in the histogramfile. As another example, an angular difference (e.g., Δθ) between thedigital image 420 and a live picture/scene displayed on the viewfinder400 may decrease as the active orientation of the camera approaches theorientation associated with the digital image 420 in the histogram file.

In some embodiments, transparent versions of multiple digital images maybe displayed on the viewfinder if the digital images were captured inrelatively close proximity to one another. FIG. 4D illustrates howtransparent versions of digital images 430, 440 are displayed on theviewfinder 400 as a position of the camera nears positions associatedwith the digital images 430, 440. As another example, FIG. 4Eillustrates how transparent versions of digital images 450, 460 aredisplayed on the viewfinder 400 as a position of the camera nearspositions associated with the digital images 450, 460.

FIG. 5 illustrates a method 500 for using augmented reality to display atransparent version of a digital image on a viewfinder of a camera asthe camera's position nears a position associated with the digitalimage. As shown, the method 500 begins with step 510, where the cameratakes a picture to capture a digital image. Next, the method 500proceeds to step 520, where a sensory module detects a position of thecamera when capturing the digital image. Thereafter, the method 500proceeds to step 530, where the detected camera position is associatedwith the digital image in a histogram file. Subsequently, the method 500proceeds to step 540, where an active position of the camera ismonitored. Next, the method 500 proceeds to step 550, where it isdetected that the active position of the camera is within a threshold ofthe camera position specified by the histogram file. Finally, the method500 proceeds to step 560, where a transparent version of the digitalimage is displayed on the viewfinder of the camera.

Aspects of this disclosure also use augmented reality to guide a user tore-configure a camera to correct or avoid image detects in a digitalimages. For instance, an instruction may be displayed on the viewfinderto reduce the flash or to re-align the camera to avoid a shadow. Inembodiments, the instructions may be determined in accordance with aquality analysis of a previous picture or image. As an example, thecamera may determine that a defect in a picture (e.g., shadow, etc.)could have been avoided had the camera position been shifted slightly,and thereafter instruct the user to shift the camera when retakingtaking the picture. In other embodiments, a real-time quality analysismay be performed on a picture scene displayed on the viewfinder to avoidthe image quality problem before the picture is ever taken.

FIG. 6 illustrates how augmented reality can be used to guide a user inre-aligning a camera. As shown, a transparent version of a digital image610 is displayed on a viewfinder 600 along with indications 620, 625that guide the user to re-align a live scene in relation to the digitalimage 610. In some embodiments, the indications 620, 625 may instructthe user to align the camera to fix an image quality issue in thedigital image 610. For example, the indications 620, 625 may instructthe user to align the camera in accordance with the “rule of thirds” sothat a horizon is positioned on or around the bottom third of the photo.The rule of thirds is a general image composition principles dictatingthat a horizon should be aligned at a line bisecting the lower third ofthe photo from the upper two-thirds. Other image composition principlesmay be used as a basis for re-aligning the camera, such as principlesrelated to headroom guidance, lead room guidance, diagonal ruleguidance, the golden ratio and others.

FIG. 7 illustrates how augmented reality can be used to guide a user inre-configuring a camera. As shown, a transparent version of a digitalimage 710 is displayed on a viewfinder 700 along with an indication 715that the digital image 710 was underexposed. The indication 715 couldinstruct the user to change any camera setting, such as the camera modeinformation (e.g., M, Av, Tv, P, Auto, Portrait, Macro, Sports, Night,Camera, etc.) aperture/shutter settings, blown out highlights, dominantblacks, and others. FIG. 8 illustrates a method 800 for using augmentedreality to guide a user to re-configure a camera to fix an image qualityproblem in a digital image. As shown, the method 800 begins with step810, where the camera takes a picture to obtain a first digital image.Next, the method 800 proceeds to step 820, where an image qualityproblem is detected in the first digital image. Thereafter, the method800 begins with step 830, where it is determined that the image qualityproblem is at least partially attributable to configuration of thecamera. Next, the method 800 begins with step 840, where the user isinstructed via augmented reality to reconfigure the camera to fix ormitigate the image quality problem. Finally, the method 800 proceeds tostep 850, where the camera re-takes the picture to obtain a seconddigital image.

Additional aspects of this disclosure provide an activity historytracking function. The activity tracking function may allow a history ofimages to be displayed via augmented reality. For instance, a user maytake a sequence of pictures at a corresponding sequence of locations,and thereafter view the sequence of pictures via the camera's displaywhile re-visiting the sequence of locations. The activity trackinghistory function may include a feature for analyzing the quality ofpreviously taken photos. For instance, history can be dynamicallyanalyzed for photo quality, and the results can be depicted usingaugmented reality.

An activity history map may be used to create a 3D model that the usercan view using augmented reality. The model is mapped to a threedimensional perspective projection that is overlaid onto a viewfinder.FIG. 9 illustrates how a 3D model of an activity history map 900 can beoverlaid onto a viewfinder. As shown, the sequence of digital images910-970 are associated with positions on the activity history map. Auser may use this 3D rendering as a guide when re-positioning the camerato view a previously captured digital image. For example, the user maymove from one location to another to view the images in a sequence. Inan embodiment, users may remove the events from the history by swipingthe overlay using one or two of their fingers, as shown in FIG. 10.

FIG. 11 illustrates a method 1100 for performing an activity trackingfunction to allow a history of images to be displayed via augmentedreality. As shown, the method 1100 begins at step 1110, where a cameratakes pictures at a sequence of location to obtain a sequence of digitalimages. Thereafter, the method 1100 proceeds to step 1120, where thecamera detects coordinates for each location in the sequence oflocations. The coordinates may relate to spatial locations andorientations of the camera when the sequence of digital images wherecaptured. Subsequently, the method 1100 proceeds to step 1130, where thecamera associates the coordinates with the sequence of digital images ina histogram file. Thereafter, the method 1100 proceeds to step 1140,where the camera monitors a position of the camera as the user retraceslocations in the sequence of locations. Next, the method 1100 proceedsto step 1150 where the camera displays the sequence of digital images ona viewfinder of the camera in accordance with the position of thecamera.

Aspects of this disclosure display a sequence of digital images in anactivity history map via augmented reality. In one example, the user maybegin by launching the camera viewfinder with no history, and thencapturing an image or video. The history may begin with the firstcaptured image/video. Previously captured images/videos may be overlaidonto viewfinder with transparency as the images are captured. As thecamera is pointed in different directions, the overlays appearsequentially on the viewfinder, with newly captured images/videos beingadded to the history as well as to the augmented reality experience. Forinstance, a one finger swipe may remove the item at the top of thestack, while a two finger swipe may remove the entire stack.

Aspects of this disclosure provide a next generation camera userinterface, which may make the camera more user-friendly. The nextgeneration interface may include an interface for viewing capturedhistory. Further, aspects of this disclosure provide merged galleryplayback with camera viewfinder experience. In some embodiments,captured images are displayed immediately in three dimensional space.Aspects of this disclosure may support panorama and photo stitchingwithout changing camera mode.

FIG. 12 is a block diagram of a processing system that may be used forimplementing the devices and methods disclosed herein. Specific devicesmay utilize all of the components shown, or only a subset of thecomponents, and levels of integration may vary from device to device.Furthermore, a device may contain multiple instances of a component,such as multiple processing units, processors, memories, transmitters,receivers, etc. The processing system may comprise a processing unitequipped with one or more input/output devices, such as a speaker,microphone, mouse, touchscreen, keypad, keyboard, printer, display, andthe like. The processing unit may include a central processing unit(CPU), memory, a mass storage device, a video adapter, and an I/Ointerface connected to a bus.

The bus may be one or more of any type of several bus architecturesincluding a memory bus or memory controller, a peripheral bus, videobus, or the like. The CPU may comprise any type of electronic dataprocessor. The memory may comprise any type of system memory such asstatic random access memory (SRAM), dynamic random access memory (DRAM),synchronous DRAM (SDRAM), read-only memory (ROM), a combination thereof,or the like. In an embodiment, the memory may include ROM for use atboot-up, and DRAM for program and data storage for use while executingprograms.

The mass storage device may comprise any type of storage deviceconfigured to store data, programs, and other information and to makethe data, programs, and other information accessible via the bus. Themass storage device may comprise, for example, one or more of a solidstate drive, hard disk drive, a magnetic disk drive, an optical diskdrive, or the like.

The video adapter and the I/O interface provide interfaces to coupleexternal input and output devices to the processing unit. Asillustrated, examples of input and output devices include the displaycoupled to the video adapter and the mouse/keyboard/printer coupled tothe I/O interface. Other devices may be coupled to the processing unit,and additional or fewer interface cards may be utilized. For example, aserial interface card (not shown) may be used to provide a serialinterface for a printer.

The processing unit also includes one or more network interfaces, whichmay comprise wired links, such as an Ethernet cable or the like, and/orwireless links to access nodes or different networks. The networkinterface allows the processing unit to communicate with remote unitsvia the networks. For example, the network interface may providewireless communication via one or more transmitters/transmit antennasand one or more receivers/receive antennas. In an embodiment, theprocessing unit is coupled to a local-area network or a wide-areanetwork for data processing and communications with remote devices, suchas other processing units, the Internet, remote storage facilities, orthe like.

While this invention has been described with reference to illustrativeembodiments, this description is not intended to be construed in alimiting sense. Various modifications and combinations of theillustrative embodiments, as well as other embodiments of the invention,will be apparent to persons skilled in the art upon reference to thedescription. It is therefore intended that the appended claims encompassany such modifications or embodiments.

What is claimed is:
 1. A method for operating a camera, the methodcomprising: monitoring an active position of the camera; and overlayinga three dimensional (3D) model over a live scene displayed on aviewfinder of the camera when the active position of the camera iswithin a threshold of a pre-defined position, wherein the 3D model isderived from a history map of one or more first images.
 2. The method ofclaim 1, wherein the pre-defined position is associated with the one ormore first images in a histogram file.
 3. The method of claim 2, whereinmonitoring a position of the camera comprises: monitoring an activelocation of the camera; and monitoring an active orientation of thecamera, wherein the active orientation includes one or a combination ofa yaw, pitch, and roll of the camera.
 4. The method of claim 3, whereinoverlaying the 3D model over the live scene displayed on the viewfinderof the camera when the active position of the camera is within thethreshold of the pre-defined position comprises: detecting whether theactive location of the camera is within a threshold distance of alocation specified by the histogram file; detecting whether the activeorientation of the camera is within a threshold angle of an orientationspecified by the histogram file; and displaying the 3D model on theviewfinder of the camera when the active location is within thethreshold distance of the location specified by the histogram file andthe active orientation is within the threshold angle of the orientationspecified by the histogram file.
 5. The method of claim 2, wherein thepre-defined position is a previous location or orientation from whichthe one or more first images was captured.
 6. The method of claim 5,further comprising: taking one or more first pictures with the camera tocapture the one or more first images; and detecting a position of thecamera when capturing the one or more first images, wherein the positionof the camera is associated with the one or more first images in thehistogram file.
 7. The method of claim 6, further comprising: detectingan image quality problem in the one or more first images; instructing auser to re-configure the camera via augmented reality to at leastpartially correct the image quality problem; and taking one or moresecond pictures after the user re-configures the camera to obtain one ormore second images, wherein the image quality problem is at leastpartially corrected in the one or more second images.
 8. The method ofclaim 7, wherein instructing the user to re-configure the camera viaaugmented reality comprises: displaying an indication on the viewfinderof the camera, the indication guiding the user to re-align the cameraprior to taking the one or more second pictures.
 9. The method of claim7, wherein instructing the user to re-configure the camera via augmentedreality comprises: displaying an instruction on the viewfinder, theinstruction prompting the user to adjust a camera setting prior totaking the one or more second pictures.
 10. The method of claim 7,wherein instructing the user to re-configure the camera via augmentedreality comprises: displaying an instruction on the viewfinder, theinstruction prompting the user to take the one or more second picturesunder different lighting conditions.
 11. The method of claim 1, furthercomprising: instructing a user to reconfigure the camera via augmentedreality by displaying instructions to re-align the camera relative tothe 3D model.
 12. A computer program product comprising a non-transitorycomputer readable storage medium storing programming, the programmingincluding instructions to: monitor an active position of a camera; andoverlay a three dimensional (3D) model over a live scene displayed on aviewfinder of the camera when the active position of the camera iswithin a threshold of a pre-defined position, wherein the 3D model isderived from a history map of one or more first images.
 13. Anelectrical apparatus comprising: a position sensor configured to monitoran active position of a camera; an electronic viewfinder configured todisplay images; and a processor configured overlay a three dimensional(3D) model over a live scene displayed on the electronic viewfinder ofthe camera when the active position of the camera is within a thresholdof a pre-defined position, wherein the 3D model is derived from ahistory map of one or more first images.
 14. The electrical apparatus ofclaim 13, wherein the pre-defined position is associated with the one ormore one or more first images in a histogram file.
 15. The electricalapparatus of claim 14, wherein the position sensor is configured tomonitor an active location of the camera, and monitor an activeorientation of the camera, wherein the active orientation includes oneor a combination of a yaw, pitch, and roll of the camera.
 16. Theelectrical apparatus of claim 15, wherein the processor is configured todetect whether the active location of the camera is within a thresholddistance of a location specified by the histogram file, to detectwhether the active orientation of the camera is within a threshold angleof an orientation specified by the histogram file, and to display the 3Dmodel on the electronic viewfinder of the camera when the activelocation is within the threshold distance of the location specified bythe histogram file and the active orientation is within the thresholdangle of the orientation specified by the histogram file.
 17. Theelectrical apparatus of claim 14, wherein the pre-defined position is aprevious location or orientation from which the one or more first imageswas captured.
 18. The electrical apparatus of claim 17, wherein theprocessor is configured to take one or more first pictures with thecamera to capture the one or more first images, detect a position of thecamera when capturing the one or more first images, wherein the positionof the camera is associated with the one or more first images in thehistogram file, detect an image quality problem in the one or more firstimages, instruct a user to re-configure the camera via augmented realityto at least partially correct the image quality problem, and instructthe user to take one or more second pictures after the userre-configures the camera to obtain one or more second images, whereinthe image quality problem is at least partially corrected in the one ormore second images.
 19. The electrical apparatus of claim 18, whereinthe processor is configured to display an instruction on the electronicviewfinder, the instruction prompting the user to adjust a camerasetting prior to taking the one or more first pictures or to take theone or more second pictures under different lighting conditions.
 20. Theelectrical apparatus of claim 13, wherein the processor is configured toinstruct a user to reconfigure the camera via augmented reality bydisplaying instructions to re-align the camera relative to the 3D model.